System and method for creating interactive panoramic walk-through applications

ABSTRACT

System and method of the invention provides for creating, storing and broadcasting interactive panoramic walk-through applications. The combination of images is determined by the user&#39;s choice of direction of displacement at each intersection point and from each view point or geographical coordinate, in order to provide a complete view from a first person&#39;s point of view. The system provides a visual perspective comparable to a human visual experience.

This application claims the benefit of U.S. Provisional Application No.61/111,346, entitled SYSTEM AND METHOD FOR CREATING AND BROADCASTINGINTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filed Nov. 5, 2008.

FIELD OF THE INVENTION

The present invention relates generally to virtual tours. Morespecifically, the present invention relates to virtual walk-throughapplications using panoramic images, 3D images or a combination of both.

BACKGROUND OF THE INVENTION

A virtual tour (or virtual reality tour) is a virtual reality simulationof an existing location, which is usually built using contentsconsisting principally of 2D panoramic images, sequence of linked stillimages or video sequences, and/or image-based rendering (IBR) consistingof image-based models of existing physical locations, as well as othermultimedia content such as sound effects, music, narration, and text. Avirtual tour is accessed on a personal computer (typically connected tothe Internet) or a mobile terminal. Although not replacing real travel,virtual tours aim at evoking an experience of moving through therepresented space. Virtual tours can be especially useful foruniversities and in the real estate industry, looking to attractprospective students and tenants/buyers, respectively, eliminating forthe consumer the cost of travel to numerous individual locations.

The word panorama indicates an unbroken view, so essentially, a panoramain that respect could be either a series of photographic images orpanning video footage. However, the terms ‘panoramic tour’ and ‘virtualtour’ are generally associated with virtual tours created using stillscameras. Such virtual tours created with still cameras are made up of anumber of images taken from a single view point. The camera and lens arerotated around what is referred to as a nodal point (the exact point atthe back of the lens where the light converges). These images arestitched together using specialist software to create a panoramarepresenting a near 360 degree viewing angle, as viewed from a single“view point”; the panoramas are each resized and configured for optimalon-line use. Some ‘panographers’ will then add navigation features, suchas hotspots (allowing the user to “jump” from one viewpoint or panoramato the next) and integrate geographic information such as plans or maps.

Current virtual tours photographic techniques suffer from severallimitations. A seamless panoramic image can not be created from stillimages whenever such still images are captured from different nodalpoints or, for two consecutive images, from a single nodal point butwith different focal lengths or focus distances. Images captured from asingle camera rotating on its nodal point can be stitched seamlessly butthis solution can not be used for applications involving axialtranslation, where, for example, images are captured from a vehicle inmotion.

Catadioptric optical systems provide images having a 360° horizontalfield of view and near 180° vertical field of view. The resultingpanoramic images are of a annular shape and generally must be sliced andopen and “unwarped” to create a panoramic image of a rectangular shape.The unwarping step causes image distortion which, together with theoptical distortions caused by the Catadioptric optics having unevenlydistributed angles along its radial axis (vertical axis of the view),must be compensated by specialised application software.

Patent document US 2007/0211955 to Pan discloses a perspectivecorrection method allowing e-panning without image distortion, whereinimage correction step is performed on image slices (horizontal sectionof the wide-angle image) by repositioning each pixel to a correspondingpoint on a cylindrical surface. This method consumes significantprocessing power and bandwidth for respectively correcting andtransmitting the images whenever fast user motion is involved duringnavigation, and is therefore not optimal for providing a seamlessnavigation experience at relatively high user directed panning speed.

Also, with current image capture solutions, objects near the camera areresponsible for occlusion on distant objects; “occlusion” meaning, withregard to 2D images, the non-projection of a surface to a point ofobservation, and with regard to a 3-D space, the effect of one objectblocking another object from view. Limitations of current virtual tourtechnology, such as object occlusion, have had the detrimental result onvirtual tours never materializing outside of the real estate industry.

Virtual Walk-Through (“VWT”) applications constitute an evolution overvirtual tours. This technology eliminates the occlusion limitation byenabling user to travel to a point where distant objects are no longeroccluded.

Commercial online walk-through products such as Google “Street View”provide virtual outdoor walk-through of cities using images capturedfrom a camera mounted on a road vehicle which circulates on motorwaysand roads at speeds ranging from 30 kmh to 80 kmh. These products arelimited to outdoor views wherein any two consecutive points of view arepositioned at a relatively long distance from each other. Typically,StreetView and applications of the like provide visualisation atroad-view level, that is, visiting a city as viewed from a car, whereinthe user follows a pathway formed by a plurality of panoramas accessiblealong main streets, the user following the pathway by “jumping”(generally from a graphical interface allowing clicking of on screenicons) from a panorama or point of view to the next distant point ofview. This application uses a series of standard photographic images,taken by multiple cameras systems mounted to produce imagesrepresentative of multiple view angles; panoramic images are produced bycomputation (stitching) of still images from the multiple cameras. Suchpanoramic images can not provide accurate representation of geometricobjects, of for example buildings, due to the inherent discontinuity(break) of such panoramic images, this discontinuity is due to thephysical impossibility of superposing a single nodal point from multiplecameras and view angles. Furthermore, “STREET VIEW” products and thelike provide images which suffer from trapezoidal distortion wheneverthe view angle is not pointing toward the horizon; this distortion isdue to the perspective. Although geometrically correct, “STREET VIEW”'simages do not reflect human vision behaviours, which keep vertical linesmostly parallel whenever a viewer tilts its view gently above or belowthe horizon.

Google “STREET VIEW” also creates ground plane distortion where planarground seems to be inclined due to the unwanted motion of the camerascaused by the inertial force. Other current walk-through products, suchas “EVERYSCAPE” (www.everyscape.com by Everyscape, Waltham, Mass.) and“EARTHMINE” (www.earthmine.com by Earthmine inc., Berkeley, Calif.) alsoproduce trapezoidal distortion, which makes them unfit for applicationsrequiring continuous undistorted images (i.e. images which more closelycorrespond to human vision), as for example, for virtual shopping.

The trapezoidal distortion drawback is inherent also to virtualwalk-through applications based on 3D virtual images, which can be usedfor example for visiting a virtual building using real-time 3D engine,such as Second Life (www.secondlife.com by Linden Research Inc, SanFrancisco, Calif.) or video games.

“EARTHMINE” provides commercial online walk-through for applicationssuch as management of buildings and assets, telemetric measurement andother cadastral works. The product combines high resolution still imagesand 3D mesh information to provide pathways wherein user jumps from onedistant view point to another.

“EVERYSCAPE” provides commercial online panoramic products whereinmotion between two consecutive view points is simulated by videopostproduction effects. This product does not allow user to pan and tiltits viewing angle during its displacement along the travel path. Duringthe travel motion effect, images are no longer panoramic unless thefields of view of images representative of the next fixed point, areconstrained to the motion axis.

In sum, current virtual walk-through applications and systems sufferseveral important limitations. Travel along pathways is achieved byjumping from one view point to another, instead of by a fluid travelmotion. Views suffer from high occlusion rate, as lots of objects arenever visible at all along pathways. Generally, images are not standardpanoramic images but rather patchy assembly of 2D images where manydiscontinuities are found on each image.

The prior art describes several techniques whose purpose is reducing thebandwidth associated with the transmission of panoramic images andapplications between a server and the user remote terminal, allowing auser to navigate a walkthrough space while downloading data.

The use of a predefined pathway has been widely adopted to preventstorage and transmission of redundant image data. Predefined pathwayshave the additional benefit of simplifying user navigation, notably bypreventing the user from searching available paths or from hittingobjects repetitively during motion, as would be the case when user triesto walk through walls or door images.

U.S. Pat. Nos. 6,388,688 and 6,580,441, both to Schileru-Rey, disclose acomputer system and method that allow interactive navigation andexploration of spatial environments, wherein pathways are represented bybranches, and intersections in the real environment are represented bynodes. User selects which path to follow from the node. A branchrepresents a video sequence or animation played during motion betweentwo adjacent view points. Virtual objects can be integrated to specificbranches or nodes, without assigning a geographic coordinate to virtualobjects; each object is linked to at least one branch or node anddisplayed when user is travelling on said branch or node.

U.S. Pat. No. 7,103,232 to Kotake discloses an IBR system with improvedbroadcasting performance, where a panoramic image is created bystitching several images from several cameras, preferably video cameras,pointing to distinct points of view, the cameras being synchronised byuse of a single time code. The '232 system provides panoramic imagesdivided in six images sections of 60° horizontal field of view, andbroadcast typically only two of the six sections' image (providing a120° field of view) at any giving time with an aim to reduce processingpower and communication bandwidth. The '232 solution is not optimizedhowever for walk through applications allowing fast movement across thehorizontal plane beyond 120°; moreover, the '232 patent does notdisclose broadcasting images of different image resolution, meaning thatit only covers broadcasting of image of the highest possible imageresolution.

U.S. Pat. No. 6,633,317 to Jiang Li discloses a data transfer scheme,dubbed spatial video streaming, allowing a client to selectivelyretrieve image segments associated with the viewer's current viewpointand viewing direction, rather than transmitting the image data in thetypical frame by frame manner. The method of patent 317' divideswalkthrough space in a grid; each cell of the grid is assigned to atleast one image of the surrounding scene as viewed from that cell;images are characterised similar to a concentric mosaic in that eachcells is represented by a sequence of image columns. Method of patent317' allows transmission of part of the images (compressed or not)needed in an attempt to anticipate viewer's change of view point withinthe walkthrough space, starting with image data corresponding toviewpoints immediately adjacent the current viewpoint, with subsequentimage data associated with viewpoints radiating progressively out fromthe current viewpoint. Patent 317' is well suited to open walkthroughspaces where user can move in any direction using multiple source ofimage data (simple 2D image, panoramic image or concentric mosaic), suchas in a typical 3D environment. However, this method is not suited tothe optimal transmission of full panoramic images in situations whereuser travels along predefined pathways consisting of several view pointsin a linear arrangement within a network of pathways within thewalkthrough space. Additionally, being view direction sensitive, thismethod is not optimized, in terms of response time, to allow user tochange his travel plan, for example, by making a U-turn or to travelalong another pathway. Finally as this method allows travel in anydirection (along a predefined pathway) the amount of data download torepresent a given view point is greater and therefore less suited forfast and responsive viewing experience on the Internet or other networkmedia having limited bandwidth.

Consequently, no system of the prior art provides a system optimized forseamless broadcasting of fluid motion where the user can orientate (panand tilt) the field of view during motion and where the user can stopthe motion anywhere along the travel path, in order to discover localobjects in detail without occlusion.

Integration of virtual objects (such as images, icons, etc) to panoramashas been limited to the integration of two dimensional objects inspecific view point images wherein said objects are not visible fromdistant view points.

U.S. Pat. No. 6,693,649 to Lipscomb discloses a solution for non-linearmapping between media and display allowing “hotspots”, defined as anoutline of two points connected by straight lines, to be used in thecontext of panoramas. Such “hotspots” are referenced to each image usingtwo angle's or two pixel's coordinates; such values are only valid foreach distinct image.

Allocation of a third dimension value for virtual objects anddetermination of precise geographic location information for each viewpoint are prerequisite for the seamless integration of hotspots andother virtual objects in panoramas, where said object would be visiblefrom any point having a direct line of sight to the object.Consequently, no prior art system provides a system and advancedfeatures based on geographical information such as the ability to pin anelement of information on any location in a view, such element stayingspatially fixed to the point during the travel. The present method andsystem for creating and assembling interactive walkthrough applicationsovercomes these shortcomings as will now be described.

Given the market need for immersive online applications, what is neededtherefore is a system and method for providing seamless, quality, fluidwalk-through navigation using any combination of 2D panoramic images,virtual 3D, and optionally, virtual objects or any images or acombination thereof.

What is needed is an optimized system or method for the real-timeconstruction and broadcasting of panoramic walkthrough applications,which allow the user, from each view point or geographical coordinatealong a network of pathway, to have a complete view from a first personpoint of view, the view covering substantially 360° in field of view.What is needed is such a system or method that combines high rate fluidpanoramic imaging broadcasting and the possibility of seamlesslyproviding higher quality images in which the visual perspectiveperception based on human vision is preserved.

What is needed is a system or method for creating and broadcastinginteractive panoramic walk-through applications that can combine stillpanoramic images that can be captured from either indoor or outdoorlocations, and virtual scenes.

Finally, what is needed is a system or method for creating andbroadcasting interactive panoramic walk-through applications that canmake genuinely interactive functions available and accessible in theimages.

SUMMARY OF THE INVENTION

A system, apparatus and method for creating interactive panoramicwalk-through applications having a 2D image acquisition system isprovided. The system includes, comprising a holding device such as avehicle equipped with a camera connected to a catadioptric opticalsystem providing near 360° field of view, a memory device adapted tostore data including images and geographic coordinate informationrelated to these images, communication device for transferring data fromthe memory device to a computer, a fixation and stabilisation systemconnecting the camera to the holding means for aligning the cameraperpendicular to the horizontal plane; a processor and associatedsoftware for performing image modification steps; and optionally, 3Dvirtual images.

The image capture system includes a location measurement device (GPS), adistance measurement device (odometer) and an inertial measurement unit(IMU) for measuring rate of acceleration and changes in rotationalattributes (attitude) of the vehicle, the fixation device or camera.

The stored data includes the image, date and time of image capture,geographical coordinate information, and other information, notablyimage reference and image group, so as to identify, for example, aparticular district or street, as well as camera settings such asaperture, and image-related information such as camera model, speed andISO reference.

The image modification steps include the steps for providing panoramicimages based on two points perspective, of unwarping images, verticalstretching of image proportional to the divergence of the field of viewto the horizon of the optical system, and expanding horizontal edges.The image modification steps optionally include the step of automaticblurring of portion of images, such as faces or numerical car plates.

Software operates a computer to perform image modification steps such asprocesses images in two resolutions, where low resolution is used forinteractive walkthrough panoramic motion, and high resolution forinteractive panoramic motion.

According to another preferred embodiment of the present invention,preservation of visual perspective based on human vision is provided,notably by use of two points perspective that does not produce thetrapezoidal distortion that is inherent with standard 360° environmentinteractive applications.

It is an object of the invention to provide a system and method forproviding immersive, interactive and intuitive walk-through applicationsusing 2D panoramic true images, virtual 3D images or a combination ofboth, that can provide seamless quality walk-through navigation and highquality imaging.

It is another object of the present invention to provide a system ormethod that combines high rate panoramic imaging broadcasting and thepossibility of seamlessly providing higher quality images in whichvisual perspective based on human vision is preserved.

It is another object of the present invention to provide a system ormethod for creating and broadcasting interactive panoramic walk-throughapplications that can combine indoor and outdoor images, based on 2Dpanoramic images and virtual 3D images.

It is another object of the present invention to provide a system ormethod for creating and broadcasting interactive panoramic walk-throughapplications that can provide genuinely interactive functions accessiblein the images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic side view of a catadioptric (mirror-based)panoramic optical system.

FIG. 2 is a schematic view of a panoramic image made using thecatadioptric based panoramic optical system of FIG. 1.

FIG. 3 is a schematic side view of a lens based panoramic opticalsystem.

FIG. 4 is a schematic view of a panoramic image made using the lensbased panoramic optical system of FIG. 3.

FIG. 5 is a flow chart showing the unwarping modification steps.

FIG. 6A-6C are schematic views of a panoramic image, made from eithersystems of FIG. 1 and FIG. 2, following each of the unwarpingmodification steps of FIG. 5.

FIG. 7 is a schematic view of a panoramic image before and aftermodification steps to compensate for vertical distortion.

FIG. 8 is a side view showing the vehicle, image capture system,measurement means and attached odometer over a schematic rulerrepresenting distance of vehicle' travel along a road.

FIG. 9 is a schematic view of the data storage apparatus of the presentinvention.

FIG. 10 is a floor plan view showing the division of a panoramic virtual3D image into four sections to be compatible with standard field of view(<179.9˜°) in standard 3D applications.

FIG. 11 is a schematic view showing four images resulting from therendering of a 360° field of view in a standard 3D application by therendering of section step, providing the division of a 360° panoramicfield of view (FOV) on four 3D images of 90° of FOV of FIG. 10.

FIG. 12 is a schematic view showing the steps of assembly and panoramicdistortion over the images of FIG. 11, wherein image 250 shows anexpansion of horizontal edges, image 252 shows a distortion of image tocompensate for standard perspective where image are projected on aplane, and image 254 shows a cut of image bulb, discontinuous section ofthe image, at top and bottom of the image.

FIG. 13 is a schematic view of a panoramic image of FIG. 7 aftermodification steps to expand horizontal edges. Same step is typicallyapplied to image 254 of FIG. 12.

FIG. 14 is a flow chart showing the two points perspective distortionsteps.

FIG. 15 is a schematic view showing the steps of modifying a panoramicimage in order to provide a resulting image based on a two pointsperspective similar to human vision.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Image Capture System

Referring now to FIGS. 1-4 and 8-9, an image capture system 20 consistsof a panoramic optic 30, 30′, a camera 40 and a memory device 50 such asa computer, mounted on a vehicle 70 or other portable holding device.

The panoramic optic 30, 30′ is a physical panoramic optic providing 2Dpanoramic images. The optic 30, 30′ including either, a “lens andmirror” based optic system (catadioptric system) 32, 38, 42 as shown inFIG. 1, or a physical optical panoramic system 33 (consisting of anultra wide angle lens or fisheye system with lens providing more than200° of continuous vertical field of view), without mirror, as shown inFIG. 3. Both systems 30, 30′ are commercially available and reflect thesubstantially 360 degree panoramic field of view into the lens basedoptics connected to camera 40.

The mirror shape and lens used is specifically chosen and disposed suchthat the effective camera 40 maintains a single viewpoint. Such a lensis available from Nikon, Canon, and other vendors for example BellissimoInc, Carlson Nev. (www.0-360.com). The single viewpoint means thecomplete panorama is effectively imaged or viewed from a single point inspace. Thus, one can simply warp the acquired image into a cylindricalor spherical panorama.

Optionally in the catadioptric system, the image capture system can usea panoramic lens optic (not shown) commercially available orspecifically designed to be adapted to the image capture system 20 ofthe present invention. This panoramic lens optic is composed of a lensassembly that distorts the field of view (FOV), wherein the FOV isexpanded (on the geometric opposite) to cover an additional 90° in alldirections, from the camera's original FOV, providing in total a FOV ofat least 180° of vertical FOV, and ideally at least 240° of verticalFOV. Such a panoramic lens allows more vertical field of view than acatadioptric system, but accentuates chromatic aberration and is morevulnerable to dust, drops and lens flare artefacts.

The most significant advantage of catadioptric systems 32 and physicaloptical panoramic systems 33 (mono camera system) over multi-cameraimage capture systems, is to provide image almost free of chromaticaberrations or discontinuities (breaks). Moreover, since a completepanorama is obtained on each image shot, dynamic scenes can be captured.

A first advantage of a physical optical panoramic and catadioptricsystem over multiple camera systems is that the former avoids the needto stitch multiple images to create a full panoramic image and imagecolor and exposure are consistent inside one point of view over the 360°range. A second advantage is that the geometric nodal point does notneed to be simulated, as is the case with stitched motion images.Moreover, with a physical optical panoramic or catadioptric system, theaccuracy of an object's geometry in the image is not relative to it'sdistance from the camera. In a multiple camera system where the nodalpoint is simulated by software techniques, objects located proximate tothe camera are discontinuous and produce ghosts images and artefactsover the resulting panoramic image.

The camera 40 used in the system 30, 30′ can be any kind of imagingdevice (e.g., a conventional camera with chemical exposition film, videocamera, etc), but is typically a high resolution digital camera, havingCCD or CMOS captors of typically 12 Megapixel of resolution or more,with controllable aperture and fast response time of typically 4.5images/second or more.

Fast response time is required to obtain still images during imageacquisition while the vehicle 70 is in motion. Speed of displacement ofthe vehicle varies; it is typically of 10 km/h and 2.5 km/h forrespectively outdoor and indoor image acquisition applications; thisprovides for a resolution of three to four or more images per meter (at2.5 km/h) for indoor applications to one image per meter for outdoorapplications (at 10 Km/h). Typical speed and number of image per meterdisclosed in this document are provided by example and do not constitutea limitation to the applicable field of the 1.5 present invention.Images can be captured at higher vehicle velocity; during which,satisfactorily images can be captured using a camera with a bettersensitivity or a lower image resolution or at a lower capture rateallowing fewer view points along a pathway. Identical or a higher numberof view points may be captured using a faster capture device. Higher orlower density (images per meter) may be achieved based on requirementsof the specific application field, and on hardware evolution.

For a low light environment, image bracketing and high dynamic range(HDR) image processing is used to enhance the dynamic spectrum of thefinal image. Image bracketing and HDR is a technique used to mixmultiple images of the same viewpoint, each image having differentexposure time. Image bracketing and HDR require the immobility of thevehicle during image acquisition, the drawback being a slower imagecapture process.

The digital camera 40 is coupled to a catadioptric optic system 32 by anoptic apparatus 42 such as is commercially available from manufacturerssuch as Nikon and Cannon, then by a standard connector 38 provided bythe catadioptric lens manufacturer for each proprietary optic mountingformat. The digital camera 40 could also be coupled with the panoramiclens 33 by an optic apparatus 43 that is commercially available from theabove-mentioned manufacturers

The memory device, such as computer 50, receives and stores the imagestransferred from the camera 40, together with other information receivedfrom measurement device 60 such as geographic coordinates (includingaltitude) related to the images as well as geometric orientation,acceleration, rate of rotation on all three axes (attitude) and traveldistance information of the capture vehicle and/or the measurementdevice. Memory device 50 is typically a computer installed with anoperating system, proprietary software and a logical device withmultiple processing cores and/or CPU arrangement. The proprietarysoftware manages the distribution of the load of image acquisition tomultiple threads on multiple CPUs, processing cores or logicalprocessing units. The distribution of load is achieved by attributingthe processing work of each subsequent image to another CPU core orlogical processing unit. In a computer 50 having four logical processingunits, a typical image capture sequence according to the presentinvention would be processed in sequential order as follows: (i) firstimage's acquisition processing work is performed on “logical processingunit 1”, (ii) second image's acquisition processing work is performed on“logical processing unit 2”, (iii) third image's acquisition processingwork is performed on “logical processing unit 3”, (iv) fourth image'sacquisition processing work is performed on “logical processing unit 4”,(v) fifth image acquisition processing work is performed on “logicalprocessing unit 1”, (vi) sixth image acquisition processing work isperformed on “logical processing unit 2”. The distribution ofacquisition processing across multiple CPUs increases the rate of imageacquisition in a given time, resulting in better image acquisitionperformance and accrued image acquisition reliability.

Images are distributed from memory device 50 to multiple storage devices52 to achieve the high data bandwidth required by the in-motion imagecapture method of the present invention. Memory device 50 and multiplestorage devices 52 are located onboard vehicle 70 or remote thereto. Acommunication device 54 allows transfer of data from the memory device50 to a central computer 80. Data is stored in a source database 400 onthe central computer 80, wherein each image has a unique identification.Each unique image ID is associated with a specific time reference beingthe database 400, the time reference is the time when the image iscaptured. Because the time reference needs high precision, the timereference is given as universal time reference, provided by the GPSunit, which is typically more precise than the internal computer clock.With the time reference, the image capture location can easily beretrieved.

Data Measurement Means

The measurement device 60 mounted on the vehicle 70 comprises a GPStracking device 62 or a similar device able to determine geographiccoordinate information from a satellite signal, radio signal orotherwise. Each image is recorded in the memory device 50 or on acentral computer 80 with the associated geographic coordinateinformation, namely of the location of image capture, which is storedeither on a dedicated recording device or on the memory device 50 suchas an on-board computer or on a remote central computer 80. Data istransferred using a communication protocol such as USB, Bluetooth,Ethernet, WiFi, and stored on the destination apparatus in any standarddatabase format.

Geographic coordinates, also referred herein as “GPS data” 162, arestored with specific GPS universal time reference to images, allowingthe determination of the exact geographic location at which each imagewas taken. Memory device 50 is synchronised to the GPS clock to storeuniversal time reference with any stored data. A system and method forthe integration of virtual objects in an interactive panoramic walkthrough application, using a method for determining geographiccoordinates with increased precision is described in PCT applicationentitled SYSTEM AND METHOD FOR THE PRECISE INTEGRATION OF VIRTUALOBJECTS TO INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, by Lindemannet al. which is being concurrently filed with the instant applicationand is incorporated by reference hereto.

Because GPS devices have limited precision in altitude tracking, otherdevices such as an altimeter or any altitude tracking device can be usein adjunction to GPS device to enhance the precision of altitudetracking of images. Further, because GPS devices have limited precisionin direction tracking, direction can be obtained from an electroniccompass or other direction tracking device, thereby enhancing theprecision of the recorded path of image.

An odometer 66 is connected to vehicle 70 for indicating distancetraveled between any two image locations, thus improving the precisionof the geographic coordinates associated with each image. Odometer 66may be an electronic or mechanical device. An Inertial MeasurementUnit—IMU device 67 on board vehicle 70 detects the current rate ofacceleration of the vehicle as well as changes in rotational attributes(attitude), including pitch, roll and yaw. Such data may be used tocorrect image inconsistencies caused by travel over uneven surfaces. Itshould be noted that the vehicle's acceleration or speed does not affectthe capture density, number of image along a pathway, because successiveimages are automatically triggered as a function of the distance valuebetween any two successive images, as provided, for example, by GPSdata, odometer data 166 and/or IMU data 167.

Vehicle

The host vehicle 70, for image capture of outdoor locations, can be anyvehicle adapted for circulation on roads, namely cars, trucks, or anyvehicle adapted to limited circulation areas or indoor circulation suchas golf carts, electric vehicles and mobility scooters (scooters for thedisabled), etc. FIG. 8 shows vehicle 70 as a small car. For typicallysmaller, steep roads as well as image capture of indoor locations,remote controlled vehicles, unmanned vehicles, robots, andstair-climbing robots, can be used as the host vehicle 70 for the imagecapture system 20. In a miniaturized version, the Image capture system20 can also be carried by a human and for some special applications,small animals such as rats. Where the terrain is difficult, a flyingmachine can be used, in which case an odometer is emulated by the use ofGPS data or/and triangulation using radio signals. Triangulationtechniques using radio signals require that two or more emitters belocated at known positions.

Logical Means for Performing Image Modification Steps

According to a preferred embodiment of the present invention, sourceimages 210 (2D panoramic images) from the image capture system 20 aremodified using a computer with a logical device such as central computer80. Images modification steps comprise the steps of unwarping 312,compensation of vertical distortion 322, expansion of horizontal edges332 and the resolution of two point perspective distortion 342, in orderto obtain release images 280 that can be broadcast by web server 82.

As shown in FIGS. 1, 3, 5 and 6A-6C, source images 210 obtained by apanoramic optic 30, 30′ are typically circular in shape, referred in theart as annular images. Unwarping 312 of source images 210 is achievedusing conventional software techniques, so as to form cylindricalimages. The unwarping operation is typically performed in threeconsecutive operations: in a first operation 313, the image is centeredand aligned relative to a grid consisting typically of geographicdirection (North-East-South-West axis) or corresponding degrees of the360 degrees panorama, where the North axis direction is referred to as 0degree for convenience; in a second operation 314, the circular shapedimage is opened up after “slicing” a section from the image center to anedge, typically the bottom point of the image on the “South” or 180degree direction the resulting image 211 is in the shape of a circulararc; and in third operation 315, the circular arc is further unwarped toform an unwarped image 212 of rectangular shape. For ease ofunderstanding, images of FIGS. 6A-6C, 7 and 13 show section marks on thevertical axis where the “A” mark indicates the South or the 180 degreedirection, “B” indicates the East or the 90 degree direction, “C”indicates the North or 0 degree direction, and “D” indicates West or the270 degree direction.

As shown in FIGS. 2 and 4, the field of view 34 of catadioptric optic istypically un-evenly distributed along the vertical axis of the mirror,causing optical deformations, where the image 212 appears compressednear the upper and lower edges of the image. To correct this verticalunevenness, compensation of vertical distortion 322 is performed asshown in FIG. 7, and involves modifying each unwarped image 212 bysoftware operation to compensate for the compression of the upper andlower edges of the images, to provide a resulting image 222 that istypically larger along the vertical axis, compared to the originalunwarped image 212.

Compensation of vertical distortion step 322 is performed by applying afunction curve to affect pixel distribution along the vertical axis. Ina first sub-step, an empty data buffer corresponding to a destinationimage 222 is set-up, resulting in a blank image. In a second sub-step,correspondence between coordinates in the source image 212 anddestination image 222 is determined using a function curve. In a thirdsub-step, pixel values (color, hue, intensity, CMYK or pixel RGB values)at given coordinates of source image 212 are copied to pixel values atcorresponding coordinates on destination image 222 on a pixel per pixelbasis. On the third sub-step, at least one image sampling method such as“median”, “summed area”, “bilinear” or “trilinear” known image samplingmethods is applied to obtain sub pixel data whenever a pixel'scoordinates, size and shape on the original image 212 do not matchresulting pixel's coordinates, size and shape on the destination image222. One resulting pixel can match a source pixel exactly or can be avariable portion of one or several source pixels), thereby allowing thedestination image to have different vertical resolution (number ofpixels) as compared to source image 212.

The function curve is determined by either measuring the curvature ofdifferent optic elements of the panoramic optic 30, typically the mainlens or mirror, or by measuring the vertical distortion produced by saidoptic elements. Vertical distortion is measured by acquiring the imageof a calibration model (a 2D image) at a given distance with the imagepanoramic optic 30 and camera 40 in a calibration room having visibledemarkations (for example, a geometrically precise grid painted on aroom wall). This helps determine distance discrepancies between theimage and the calibration model. In other words, on the destinationimage 222, a measurement unit (for example 1 vertical meter) at a givendistance (for illustration purpose a measurement unit of 1 verticalmeter at the distance between a tree, 213 on both images 212 and 222,and the position of the capture optic 30) fills the same number of pixelnear the horizon and near the top or bottom of the image. Thismeasurement unit which fills a number of pixels N near the horizon ofthe image, will fill the same number of pixels N at the top or bottom ofthe image. On the unwarped image 212, same measurement unit at samedistance will fill fewer pixels near the top and bottom of imagecompared as to the horizon (image vertical middle).

Use of Virtual 3D Images Instead of Real Camera Image.

The system 20 of the present invention also allows the use of 3D virtualimages, alone or in combination with 2D (optical) panoramic images, forthe purpose of creating and broadcasting interactive panoramicwalk-through applications. Current commercially available 3D renderingsoftware engines used in CAD (“Computer aided-design”), 3D modeling andgaming applications are not meant to provide panoramic images havingnear 360 degree horizontal field of view. As shown in FIGS. 10 to 15,the computation of a virtual 360 degree panoramic image is thustypically achieved by transforming a 3D scene 240, originating from a 3Dmodeling, for example, into a panoramic 254 image equivalent to theresulting image 222. Because standard 3D software is not meant to renderimage that have an horizontal field of view larger than ˜179.9°, thecomputation of a virtual 360 degree panoramic image is thus achieved bydividing the 360° field of view in 4 sections of 90° fields of view,which in FIG. 10 is represented by numbers 1, 2, 3 and 4.

According to a preferred embodiment of the present invention purpose isto increase image broadcasting speed and overall system performance,computing a virtual 360 degree panoramic image is performed, for each 3Dscene 240, in a first step 253 by rendering four images, called sections244 (numbered 1 to 4 on each of FIGS. 10-12 and 15), having a 90°horizontal field of view from a single nodal point 241 and each section244 pointing toward one of the four right angle directions (front, left,back and right). From the nodal point 241 each section 244 is thusoriented, at a view angle of 90°, from the preceding and followingsections. The combination of the four original sections 244 can berepresented by a single rectangular shape image 250. In 3D space, thecombination of the four 90° view sections can be used to define theinner faces of a box; consequently, image 250 represents thejuxtaposition over the same plane of the four inner faces of a boxviewed from a central position inside this box.

As disclosed by US 2007/0211.955 to Pan with respect to panoramicimaging, and as is known in 3D imaging, the projection of wide angle orpanoramic images over a cylindrical field of view reduces the number ofcomputing steps necessary to provide real-time compensation to imagedistortion, and thus eases the creation of interactive panoramic motion.This is due to the fact that, in this kind of image, each image sectionhas the same proportion ratio independently of the angle of viewingangle.

In a second step, according to a preferred embodiment of the presentinvention, as shown in FIG. 12 image 250 is modified as if image 250 wasthen projected over a continuous surface of cylindrical shape. Thisimage modification is done by stretching vertically the resulting image250, section per section, along a sinusoidal curve using software whichaffects the sinusoidal stretching logical step 352, and resulting inmodified image 252. This sinusoidal stretching logical step 352 isdescribed in detail below.

In a first sub-step an empty data buffer corresponding to thedestination image 252 is setup (resulting in a blank image).

In a second sub-step, the sinusoidal stretching logical step 352 isperformed by evaluating resulting images pixels using a sinusoidalfunction curve that points on the original image pixels. This functioncurve computes the vertical coordinates in the destination image 252 todetermine the corresponding coordinates in the source image 250. Thissinusoidal function curve decreases its amplitude for verticalcoordinates that are farther from the top edge of the section, theamplitude reaching a null value (0) at the vertical center of thesection, then the amplitude continues to decrease as negative number andreaches the exact opposite value at the bottom edge of the section. Thissinusoidal function curve is calculated from the start of an imagesection 244 to the end of the same section. The same function curve isused horizontally for each section, which results in an image plane 252that contains 4 sections stretched in the same way regarding to thesection local coordinate system.

In a third sub-step, pixel values at a given coordinate of source image250 are copied according to pixel color value at correspondingcoordinates (given by said sinusoidal function curve of the second substep) on the destination image 252, on a pixel per pixel basis. Duringthe third step, at least one image sampling method such as a, “median”,“summed area”, “bilinear” or “trilinear” known image sampling method isapplied to obtain sub pixel data, whenever the pixel's coordinates, sizeand shape, on the source image 250 do not match the resulting pixel'scoordinates, size and shape on the destination image 252 (one resultingpixel can match a source pixel exactly or can be a variable portion ofone or many source pixels), allowing the destination image 252 to havedifferent vertical resolution (number of pixels per mm) compared tosource image 250. Pixel values can be either color, hue, intensityvalues, CMYK (Cyan, Magenta, Yellow, Key) values or pixel RGB (Red,Green, Blue) values.

The sinusoidal stretching step 352 is performed to compensate thedistance difference (i) between the view point and the pixels laid overa flat surface (the face of a box discussed above) and (ii) between theview point and same pixel laid over a cylindrical surface (the innersurface of a cylinder as viewed from within). In geometric perspective,the area, filled with the projection of an object, grows in the inverseproportional manner as the distance between said object and the viewpoint. The center of an edge of a flat surface (any face of the box) islocated closer to the viewpoint, compared to the center of an edge of anequivalent cylindrical surface (as if the box would be contained withinthe arc of a cylinder). In the geometric projection of a box to acylinder, the upper and lower edges of each face of a box projected fromthe common center of the box and the cylinder become an arc that followa portion of a sinus curve. Any pixel of the image is then positioned inthe resulting image 252 relative to its vertical distance between saidpixel and top and bottom edge of the box, at the same relative distanceto the top and bottom edge (arc) of the cylindrical projected box. Topand bottom edges of the projected faces of the box to a cylinder arecomparable to top and bottom edges in the modified image 252.

In a third step, a logical step 354 is performed which removes thecircular area on the upper and lower edges of each image's section ofimage 252, so that ultimately a single rectangular shape image 254 iscreated.

In order to increase image broadcasting speed and overall systemperformance, each image 222 (at this stage, image 254 and image 222 canbe processed by the system in the same manner), which is either apanoramic image from the image capture system 30 or an image resultingof the modification of a 3D image, is further modified by the expansionof horizontal edges logical step 332 shown in FIG. 13. Image 222 istypically divided in two sections. The left edge of image 222 iscropped, copied and pasted on the opposite edge of the image to formexpanded image 232. In other words, the left border section of eachimage is sliced in a sub-section on the vertical axis, and the edgesub-section is cropped and displaced on the opposite edge of the image,so that two identical sub-sections of the image appear on both oppositesides of the image. The expansion of horizontal edges step 332 reducesthe processing workload necessary for the real-time performance of thetwo points perspective distortion step 342, allowing an output ratetypically of 15 or more images per second. This rate is sufficientlyfast to create the illusion of a seamless motion, without jerkymovements, during displacement (image panoramic motion or rotation ofthe view) within the image. Increase of system performance is achievedbecause the expansion of horizontal edges 332 avoids need for the imagebroadcasting system to manipulate two images simultaneously. Otherwise,this is the case every time the interactive panoramic reaches a verticalimage split at 0° or 360° image borders. The system of the presentinvention achieves perpetual panoramic motion without slowdown: when theborder of an image is reached, the image broadcasting system simplydisplays the other side of the same image, in an invisible manner to theuser.

A virtual walk-through application providing a convincing natural,immersive navigation environment should be visually as close as possibleto human vision during panoramic motion (view point rotation, horizontaldisplacement inside one panoramic image), or translation motion (viewpoint moving in space forward or backward on the pathway), irrespectiveof motion speed.

According to a preferred embodiment of the present invention, apanoramic image closer to human vision during panoramic motion, withoptimised calculation performance is provided by the use of two-pointperspective distortion. During vertical panoramic motion, the image isvertically compressed depending on vertical elevation (distance to thehorizon) in order to compensate for significant horizontal distortionswhich occur further from the horizon. Such distortion is systematicallycaused by the cylindrical projection of the image and is more severetoward the vertical lower and upper parts of the image. The presentinvention provides compensation of horizontal distortion similar to theprojection of a cylindrical view to a planar view. In order to speed upimage modification steps to real time, the present invention departsfrom the traditional projection of a plane inside a cylinder byassembling an image from the juxtaposition of several vertical imagestrips 235, each of them optionally having a different width, and bystretching said strips. This avoids the need to recalculate the positionof each pixel of the image according to the projection of a plane insidea cylinder. Only a selected vertical and a horizontal portion of theimage are displayed and visible to the user at any given time. Themethod of the present invention described below works in a“staircase-like” manner (visible on the zoomed image 236 of FIG. 15)whereby the width of each stripe is controlled and reduced and thenumber of stripes increased to provide the illusion of a progressivecurvature that is a perspective closer to human vision.

For this purpose, the panoramic images 232 are further modified by thetwo points perspective distortion step 342 in order to provide imagesbased on a two points perspective. The two points perspective distortion342 step is achieved by applying two basic operations, as shown in FIG.15, allowing vertical parallel lines in the real world to appear asparallel lines on screen during the panoramic motion and interactivenavigation.

In a first operation 343 (shown in FIG. 14), the viewable portion 234 ofany image 232 viewed by a user on screen, is divided into verticalslices 235 (stripes) in real time, which are each vertically stretchedalong sinusoidal lines in real time to recover a two points perspective.The sinusoidal virtual canvas (the vertical slices) is in a fixedposition relative to the viewport which is the rectangular window bywhere the image is viewed by the user. As the user pans his view duringnavigation, the viewable portion 234 travels across fixed verticalslices 235, during which time only a portion of the panoramic image isvisible on screen.

In a second operation 345, vertical panoramic interactive motion isachieved by simple vertical translation of vertical slices 235 withoutaffecting sinusoidal distortion (that is the relative position andrelative size of each stripe regarding each other stripe). All theslices 235 are then vertically compressed proportional to the angulardivergence of the field of view to the horizon, in order to minimisevertical distortion of the two points perspective. The two pointsperspective distortion 342 avoids trapezoidal distortion found on aregular three point perspective used in 3D computer graphics.

The image modification steps of the present invention processes imagesin two resolutions; low resolution for interactive walkthrough panoramicmotion (view point translation), and high resolution for interactivepanoramic motion (view point rotation) applications.

Additional image modification steps are possible, through the use of alogical processor.

In order to ensure protection of privacy and to avoid infringement toprivacy rights and right to images, the system of the present inventioncan include logical means for the automatic blurring, masking or removalof objects appearing on images, such as human faces and car plates.Automatic blurring of faces is achieved by a face recognition algorithm.Using image tracking software known in the art, each individual face isidentified and tracked throughout any sequence of images. Based on thelocation of faces or image-based coordinates provided by the trackingsoftware mechanism, software blurs the images on the coordinatescorresponding to recognizable faces. Automatic blurring of car plates isachieved by a car plate recognition algorithm. Using image trackingsoftware known in the art, each individual car plate is identified andtracked throughout any sequence of images. Based on the location of carplates or image-based coordinates provided by the tracking softwaremechanism, a software blurs car plates on the coordinates correspondingto recognizable car plates. Identical processes can be used to maskobjects from images. The processes above are not limited to faces or carplates but can be used to blurred or mask any objects specified by thesystem's operator, with corresponding software adaptation.

Management of Images

According to a preferred embodiment of the present invention, panoramicresulting images 232 are stored on the central computer 80, each imagebeing associated with tridimensional (XYZ) geographic coordinatesindicative of the image's location of capture and optionally otherreferences such as date of capture, with day and time; project or nameof location; image group, so as to identify for example a particulardistrict or street; digital photography settings, such as aperturesetting, speed, ISO (measure of the light sensitivity of the digitalsensor), exposure, light measure, camera model; identified for mountedphotos filters such as UV, neutral gray, etc.; vehicle information suchas vehicle speed and model; and camera operator identification.

Precise reference to the geographic coordinates indicative of theimage's location of capture allows, among other things, the combinationor the superposition of panoramic images of the present invention withany other heterogeneous geo-referenced digital information. Views of astreet can therefore be combined with data of a GIS (“GeographicInformation System”) application, such as, for example, layers of theutilities' services infrastructure (water, cable, electric distributioninfrastructure), or contact information associated with commercialestablishments located in the vicinity, PCT application No. ______, toLindemann et al, entitled SYSTEM AND METHOD FOR CREATING ANDBROADCASTING INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filedconcurrently with the instant application and incorporated by referencehereto, discloses a method for the assembly and broadcasting ofinteractive panoramic walk through applications, using an object-baseddatabase. Still further, PCT application No. ______, to Lindemann et al,entitled SYSTEM AND METHOD FOR THE PRECISE INTEGRATION OF VIRTUALOBJECTS TO INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filedconcurrently with the instant application and incorporated by referencehereto, discloses a method for the integration of virtual objects tointeractive panoramic walk through applications, using a method fordetermining geographic coordinates with increased precision. Theforegoing PCT applications provide functionality which enhances the userexperience or increases the efficiency of the system 20 of theinvention.

3D Computer Graphic Layering

According to a preferred embodiment of the present invention, the systemand method for creating panoramic walk-through applications hereinprovide image modification steps that enable the seamless andtransparent integration of panoramic images (from a panoramic optic 20)with virtual (3D) images, wherein virtual images and panoramic imagescan be combined in a plurality of ways, for example, to mask orsuper-impose each other. It is therefore possible to combine panoramicimages, virtual images and 3D objects without limitation in anypanoramic view; the resulting view having a seamless appearance duringwalkthrough interactive motion and panoramic motion.

The system of the present invention positions each image and virtualobjects on a 3D space using absolute geographic coordinates. Anypanoramic image is edited from either a source image 210 or a virtualimage 240 has geographic coordinates in 3D space. Due to the use ofabsolute coordinates, the system of the present invention allows theaddition (pining) of virtual 3D objects to any panoramic image orvirtual image.

Masking operations are achieved by the system in which a mask isdefined, that is, the representation in shape and volume of a realobject appearing on the panoramic images is defined. This technique iswell known in the art and used to integrate 3D objects in photos in thereal-estate 3D industry, wherein said technique is limited to staticview. Effectively, the mask position and orientation are correct for agiven 3D object only for a given view point.

According to an embodiment of the present invention, the use of absolute3D space coordinates for referencing both panoramic images and virtualimages inside the 3D space allows any given 3D mask to be used on anypanoramic image in a pathway. The mask becomes visible from any viewpoints that have a view on the mask's geographical coordinates.

The 3D mask is designed to exist at a position and a shape identical torespectively, the 3D position and shape of a real physical objectviewable on one or more panoramic images. When the 3D mask applied tocover a real physical object visible on a panoramic image is locatedcloser to the viewpoint of the panoramic image, compared to the 3Dobject (i.e., located further from the viewpoint), the 3D mask pixelswill occlude the pixel of the 3D object; in which case, pixels of thepanoramic image representative of the physical object may become visiblewhen the mask is rendered (the mask is “transparent” to an extent wherepixels of the 3D object behind the mask pixels also become transparent,revealing the pixel of background panoramic image). In some cases, themask can be displayed with a virtual shadow from, or cast virtual shadowto, other 3D objects to enhance the realism of the integration of 3D inthe real image.

A 3D mask can be occluded by a 3D object when the pixels representingsaid 3D object are located, relative to the view line, in front (closerto viewpoint) of pixels of said mask. As the masking process isdetermined on a pixel per pixel basis, complex situations may arise. Forexample, a 3D object can be partially occluded by a 3D mask that ispartially occluded by another 3D object. The flexibility of the maskingprocess allows a near visually perfect integration of 3D object insidereal image used in interactive panoramic walk-through applications.

According to a preferred embodiment of the present invention, 3Dcomputer graphic layering using masking operations will produce newpanoramic 3D layered images based on real world panoramic image composedof pixels from 3D objects and 3D masks. These panoramic 3D layeredimages are stored in the database 400 with geographic coordinates of the3D space equivalent to coordinates of the source panoramic image andtagged with a reference to the 3D computer graphic layering program.Said 3D computer graphic layering program reference can be a 3Dapplication source, or a file name, a project name or an applicationname or both. 3D computer graphic layering can be used to mix 3D imagesfrom different 3D rendering packages, or to add objects in the 3D spacewithout the need to re-render the full 3D panoramic images (typicallynecessary when inserting new objects in already rendered 3D panoramicimages). 3D computer graphic layering can therefore be used forinsertion and management of advertizing, wherein a virtual advertisementcan be inserted by defining a 2D (flat) or 3D object and superimposingan advertising image on said object.

Image is available in alternative version broadcasted depending on thecontext: typically, one panoramic representation of a location is editedwithout any advertising, and several other copies are made available todifferent advertisers; a software based management tool allowsadvertisers to select an advertising to be imported in the system;selection and display of advertising to be broadcast can be dependentupon, for example, available user information such as location andlanguage, and the identity of the web portal from which the user isaccessing the panoramic application.

Other characteristics and modes of execution of the invention aredescribed in the appended claims.

Further, the invention should be considered as comprising all possiblecombinations of every feature described in the instant specification,appended claims, and/or drawing figures which may be considered new,inventive and industrially applicable.

Multiple variations and modifications are possible in the embodiments ofthe invention described here. Although certain illustrative embodimentsof the invention have been shown and described here, a wide range ofmodifications, changes, and substitutions is contemplated in theforegoing disclosure. While the above description contains manyspecifics, these should not be construed as limitations on the scope ofthe invention, but rather as exemplifications of one or anotherpreferred embodiment thereof. In some instances, some features of thepresent invention may be employed without a corresponding use of theother features. Accordingly, it is appropriate that the foregoingdescription be construed broadly and understood as being given by way ofillustration and example only, the spirit and scope of the inventionbeing limited only by the claims which ultimately issue in thisapplication.

1. A method for creating panoramic images for use in walk-through applications comprises the steps of: (a) capturing panoramic images using a panoramic optic system (30, 30′); (b) unwarping images, comprising the sub-steps of: (i) centering and aligning the image (210) relative to a grid; (ii) opening up the images after slicing a section from the image center to an edge to form an image (211) having a circular arc shape; and (iii) unwarping the circular arc shape image to form an unwarped image (212) of rectangular shape; (c) vertical stretching (322) of the image proportional to the divergence of the field of view to the horizon of the optical system; and (d) expanding horizontal edges (332), comprising the sub-steps of (i) cropping and copying a section of the image (222) located on the left or right edge of the image; and (ii) pasting said section on the opposite edge of the image to form an expanded image (232), so that two identical sub-sections of the image appear on opposite sides of the image.
 2. The method of claim 1 wherein the vertical stretching of image step (322) comprises the sub-steps of: (i) setting up a data buffer corresponding to a destination image (222); (ii) using a function curve to determine correspondence between coordinates in the source image (212) and destination image (222); and (iii) copying pixel values of source image (212) to pixel values at corresponding coordinates on destination image (222) on a pixel per pixel basis, using an image sampling method selected from a group consisting of “median”, “summed area”, “bilinear” and “trilinear” image sampling methods.
 3. A method for computing a virtual 360 degree panoramic image from a virtual 360 degree scene (240), for use in walk-through applications, comprising the steps of: (a) rendering (253) the scene in four sections (244), each having a 90° horizontal field of view, each section being oriented, from the nodal point (241), at a view angle of 90°, from the preceding and following sections, said sections being represented in a single image (250) of rectangular shape; (b) vertically stretching (352) the single image, section per section, along a sinusoidal curve; and (c) removing (354) upper and lower edges sections to provide a resulting image (254) of rectangular shape.
 4. The method of claim 3, further comprising the additional step of expanding (332), horizontal edges comprising the sub-steps of: (a) cropping and copying a section of the image (254) located on the horizontal edge of the image; and (b) pasting the section on the opposite edge of the image to form an elongated image (232) which then comprises the section on both of its opposite sides.
 5. The method of claim 3 wherein the vertical stretching step (352) comprises the sub-steps of: (i) setting up a data buffer corresponding to a destination image (252); (ii) sinusoidal stretching (352) using a function curve to compute the vertical coordinates in the destination image (252) to find the corresponding coordinates in the source image (250); and (iii) copying pixel values of source image (250) to pixel values at corresponding coordinates on destination image (252) on a pixel per pixel basis, using an image sampling method selected from a group consisting of “median”, “summed area”, “bilinear” and “trilinear” image sampling methods.
 6. The method of claim 2 wherein the pixel values comprise values selected from a group consisting of “Hue, Lightness, Saturation values”, “CMYK values” and “pixel RGB values”.
 7. A method for computing a virtual 360 degree panoramic image based on two points perspective, for use in assembling and broadcasting of walkthrough applications, providing panoramic images closer to human vision during panoramic motion, comprising the steps of: (a) dividing the viewable portion (234) of a given panoramic image (232), in vertical slices (235); (b) stretching said vertical slices (235) along sinusoidal lines to recover a two points perspective, wherein a group of vertical slices is in a horizontal fixed position relative to a viewport; (c) providing vertical panoramic interactive motion by vertical translation of said vertical slices (235) without affecting sinusoidal distortion, wherein said group of vertical slices are vertically compressed proportional to the angular divergence of the field of view to the horizon, in order to minimise vertical distortion of two points perspective.
 8. The method of claim 7, wherein said vertical stripes are of non-uniform width.
 9. The method of claim 7, wherein said steps are performed in real time and broadcasted by a web server (82) to a user terminal, in response to user motion in a walkthrough space.
 10. The method of claim 1, wherein the image modification steps further include the step of automatic blurring of portion of images, such as faces or numerical car plates.
 11. The method of claim 1, wherein image modification steps processes images in two resolutions; low resolution for interactive walkthrough panoramic motion, and high resolution for interactive panoramic motion.
 12. A system implementing the method of claim 1, comprising (a) an image capture apparatus (20) for capturing panoramic images, further comprising a camera (40), an optic system (30, 30′) connected to the camera, the optic system providing a substantially 360° field of view, and a memory device (50); (b) a communication device for transferring data from the memory device to a computer (80); (d) instructions executable on a computer for performing image modification steps; and (e) a database for storing image related data.
 13. The system of claim 12, wherein the image capture apparatus further comprises a memory device to store data including images, geographic coordinate information related to said images, and distance information.
 14. The system of claim 12, wherein the image capture system comprises a location measurement device (GPS) and a distance measurement device.
 15. The system of claim 12, wherein the image capture system comprises an inertial measurement unit for measuring rates of acceleration and changes in rotational attributes of the vehicle, fixation means or camera.
 16. The system of claim 12, wherein the image related data includes the image, date and time of image capture, geographical coordinate information, project or group related information, so as to identify for example a particular district or street, and camera settings such as aperture, speed and ISO.
 17. The system of claim 1, wherein the computer which executes instructions for performing image modification steps, processes images in two resolutions; low resolution for interactive walkthrough panoramic motion, and high resolution for interactive panoramic motion.
 18. The method of claim 5 wherein the pixel values comprise values selected from a group consisting of “Hue, Lightness, Saturation values”, “CMYK values” and “pixel RGB values”.
 19. A system implementing the method of claim 2, comprising (a) an image capture apparatus (20) for capturing panoramic images, further comprising a camera (40), an optic system (30, 30′) connected to the camera, the optic system providing a substantially 360° field of view, and a memory device (50); (b) a communication device for transferring data from the memory device to a computer (80); (d) instructions executable on a computer for performing image modification steps; and (e) a database for storing image related data.
 20. A system implementing the method of claim 3, comprising (a) an image capture apparatus (20) for capturing panoramic images, further comprising a camera (40), an optic system (30, 30′) connected to the camera, the optic system providing a substantially 360° field of view, and a memory device (50); (b) a communication device for transferring data from the memory device to a computer (80); (d) instructions executable on a computer for performing image modification steps; and (e) a database for storing image related data. 